Field of Invention
Various embodiments of the present invention relate to a technology of performing the step of shuffling and merging map results at a distributed shared storage level so as to improve the bottle neck problems in the performance of a map/reduce model, thereby providing a high-speed analyzing performance.
Description of Related Art
Due to the digital innovation accelerated by the development of information communication technologies, in the modern society, massive data is being created in real time. The emergence of various information channels represented by social service and smart devices and the increase of production, distribution, and capacity of information due to such emergence of various information channels led to a significant increase of data, exceeding the limitations of conventional data management and analyzing systems.
A map/reduce model is a representative open software based framework for storing and quickly analyzing such rapidly increasing data. The biggest characteristic of the map/reduce model is in dividing the data into small pieces and then distributing the data in a plurality of cluster nodes, so that calculation operations for analyzing the data can be performed in the nodes where the data is stored, thereby maximizing the parallel processing effect. However, in conventional map/reduce models, performance deteriorates severely due to excessive costs incurring at the step of shuffling and merging map intermediate results, which will be explained hereinafter with reference to FIGS. 1 and 2.
FIG. 1 is a view illustrating a concept of an operating process of a conventional map/reduce model, and FIG. 2 is a view illustrating a merging process of a reduce node in the conventional map/reduce model.
Referring to FIG. 1, an analyzing process performs a total of five steps as follows.                Step 1: reading data to be analyzed from a shared file system 101 and transmitting the data to a map function 102 input.        Step 2: storing intermediate data analyzed in each map function 102 and in a local disk 103 of a node where the map function 102 is performed.        Step 3: shuffling step of transmitting the intermediate data stored in the local disk 103 of all nodes to a reduce node through a network.        Step 4: receiving an intermediate result of each map function 102 from the reduce node via the network, and merging the intermediate result into one file so as to transmit the file to the reduce function 106.        Step 5: reading, by each reduce function 106, the merged file to output a final result, and recording the final result in the shared file system 101.        
What the present invention noted is that in the aforementioned model, step 3 (shuffling among nodes) and step 4 (merging intermediate data) unnecessarily waste the network and disk resources, thereby causing deterioration of the entire analyzing performance.
First of all, at step 3, data transmission occurs from all map nodes to all reduce nodes in a method so called “All-to-All Shuffle”, and thus extreme network resource consumption is inevitable. If there are M number of map nodes, and N number of reduce nodes, network transmission over the entire servers will occur for M×N number of times, resulting in consuming all the network resources.
Secondly, the step of merging the intermediate result received from each map function 102 at step 4 into one aligned file in order to transmit the same to the reduce function 106 is also being pointed out as a big cause behind deteriorating the entire analyzing performance. The reason to this will be explained with reference to FIG. 2.
As can be seen from FIG. 2, the reduce node receives the intermediate result of each map function 102 via the network 202, but cannot transmit the intermediate result directly to the reduce function 106, but has to temporarily store it in separate files 203 on a local disk 105.
The reason why the reduce node cannot transmit the data received via the network 202 directly to the reduce function 106 is because the results 203 of each map function 102 are aligned only at the level of each File-A, File-B, and File-C, and not aligned as a whole.
Therefore, the reduce node 106 reads each of File-A, File-B, File-C 203 temporarily stored in the local disk 105, goes through a merging operation in the memory 204, and re-records the ultimately aligned data in the local disk 105 as a separate File-M1 205, and then removes File-A, File-B, File-C 203 from the local disk 105.
When the merging operation is completed, the reduce function 106 reads the merged File-M1 file 205 from the local disk 105, and generates the final result.
As a result, in order to perform one reduce function, for the data of the size of the map intermediate result 203, twice the amount of WRITE and twice the amount of READ occurs through the local disk 105. This example is based on an assumption that there are three output results from the map function, but in actual large volume analyzing environments, there are many cases where one reduce node processes more than several dozens of map results, in which case the cost for temporarily storing, reading, merging and removing more than several dozens of intermediate data in and from a local file system will become a big cause behind significantly deteriorating the entire analyzing performance.